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Abstract. If G is a (connected) complex Lie Group and Z is a generalized flag 
manifold for G, then the open orbits D of a (connected) real form Go of G form 
an interesting class of complex homogeneous spaces, which play an important role 
in the representation theory of Gq. We find that the group of automorphisms, i.e., 
the holomorphic diffeomorphisms, is a finite-dimensional Lie group, except for a small 
number of open orbits, where it is infinite dimensional. In the finite-dimensional case, 
we determine its structure. Our results have some consequences in representation 
theory. 



§1. We determine the automorphism groups for a certain interesting class of com- 
plex homogeneous spaces. Denote by Z a generalized flag manifold for a connected 
complex semisimple Lie group G. A real form Go (which we assume to be con- 
nected) of G acts on Z with a finite number of orbits, thus there are always open 
orbits (cf. [22]). These open orbits play a key role in the representation theory of 
Gq. An open Go-orbit D in Z has a Go-invariant complex structure. The identity 
component of the group of holomorphic diffeomorphisms of D will be denoted by 
Hol(D). In the main theorem below we determine Hol(D) for each measurable 
open orbit (see Definition 2.1). In the case where D is measurable, D carries a 
Go-invariant (usually) indefinite hermitian metric and we determine its group of 
hermitian isometries. Generally the open orbits D are non-compact, however, our 
results include the cases where Go is a compact real form and D is compact, so 
D = Z. The compact case is contained in [12], [21], [3] and [2], from various points 
of view. 

In general, for a complex manifold X, Hol(X) is a (finite-dimensional) Lie group 
if X is compact and may or may not be a Lie group if X is non-compact. For 
example, Hol(C n ) is infinite dimensional. Our main interest is when Go (so D) is 
non-compact. We give a precise condition for Hol(D) to be a Lie group. 
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Our main results are contained in the following theorem. 

Main Theorem. Suppose G is a connected simple complex Lie group and D is an 
open measurable Go-orbit in a generalized complex Hag manifold for G. 

(1) If there is a Go-equivariant holomorphic fibration of D over the hermitian 
symmetric space for Go (and D is not equal to the hermitian symmetric 
space itself), then Hol(D) is not a finite-dimensional Lie group. 

(2) If no such fibration exists, then Hol(D) is a Lie group and, except for the 
cases listed in Table 1.1 below, we have: 

(a) Hol(D) = Go if Gq is non-compact and, 

(b) Hol(D) = G if Gq is compact. 



Go 


z 


Hol(D) 


SO e (2p, 2q + l),p^0 


{pure spinors in C 2p+29+1 } 


SO e (2p, 2q + 2)/Z 2 


SO(2n + l) 


n 


SO(2n + 2, C)/Z 2 


Sp(n, R) 


Qp2n— 1 


SU{n,n)/Z 2n 


Sp(f>, q), pq + 


11 


SU(2p, 2q)/Z 2p+2q 


Sp(n) 


11 


SL(2n,C)/Z 2n 


G2, split 


quadric in CP 6 


SO e (3,4) 


^2, compact 


1 1 


SO(7,C) 



D is any open orbit in Z. 
Table 1.1 



(3) The group of hermitian isometries (i.e., the group of holomorphic diffeomor- 
phisms preserving the hermitian metric) is 

(a) Hol(D) if G is non-compact and, 

(b) a compact real form of Hol(D) if Go is compact. 

In Proposition 3.11 we will see how the case of a semisimple group reduces to 
the case of simple groups. 

The method of proof is to study the Lie algebra of global holomorphic vector fields 
on D using some standard techniques from representation theory. In most cases, 
this Lie algebra is just g. However, in other cases, it is a bigger finite-dimensional 
Lie algebra jj 1 . In each of these cases we find a group Gq which has (complexified) 
Lie algebra 1 and has an effective action on D. 

Our results have several consequences for the representations associated to the 
open orbits. In the cases listed in Table 1.1 we view Gq as acting on D. The 
irreducible representations occurring in Dolbeault cohomology spaces on D extend 
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to (irreducible) representations of G\. Also, the results have implications for a 
space of maximal compact subvarieties of D, which in turn plays a role in certain 
realizations of these representations. This will be discussed in the final section. 

We are grateful to T.N. Bailey for helpful discussions regarding Proposition 2.3. 
and especially to D.A. Vogan for suggesting the approach to Propositions 2.4 and 
2.5. 

§2. Some detailed information is obtained on the Lie algebra of global holomorphic 
vector fields on D. It will follow that Hol(D) is usually finite dimensional and its 
structure will be narrowed down to a few possibilities. We start with some notation. 

Let G be a connected simple complex Lie group. As mentioned in the intro- 
duction, the semisimple case can be reduced to the simple case. Fix a generalized 
complex flag manifold Z for G. Then, Z is (biholomorphic to) G/Q where Q is 
a parabolic subgroup of G. We follow the common practice of denoting the Lie 
algebra of a Lie group by the corresponding gothic letter. Thus, the Lie algebras 
of G and Q will be denoted by g and q, respectively. A connected real form of G 
will be denoted by G , with Lie algebra g . By Theorem 2.6 of [22], G acts on Z 
with a finite number of orbits. Hence we know that open orbits always exist. Fix a 
Cartan involution 9 and let K (respectively, Kq) denote the fixed-point group of 6 
in G (respectively, Go). Fix an arbitrary open orbit D in Z. For zq G D, Stabc(^o) 
is a parabolic subgroup Q. The Levi decompositions are given by Q = LU and 
q = [ + u. By Theorem 4.5 of [22] we may choose zq so that q contains a Cartan 
subalgebra f) of g, so that t = f) n t is a Cartan subalgebra of t. 

Definition 2.1. The orbit D is measurable if and only if D carries a Go-invariant 
measure. 

By Theorem 6.3 of [22], D is measurable if and only if q D q = I. This is 
equivalent to the statement that Stabc (zo) = L , a real form of L. We describe 
this in terms of the Lie algebras as follows. Let A = A(f),g) be the roots of f) 
in g. Then, there exists A G Uq such that A(h, [) = {a G A | (A , a) = 0} and 
A(h,u) = {a G A | (Ao, a) < 0}. Also, u is the complex conjugate (with respect to 
the real form g ) of a subalgebra u and q = [ + u is the parabolic opposite to q. We 
assume from now on that D is a measurable open 67o-orbit in Z. 

Remark. It is the measurable open orbits that play an important role in represen- 
tation theory. If Go contains a compact Cartan subgroup or if Z is the full flag 
manifold (so Q is a Borel subgroup), then all open orbits are measurable. Also, if 
one open 67 -orbit in Z is measurable, then all open orbits are measurable. A simple 
example of a non- measurable open orbit is the (unique) open orbit of SL(3, R) on 
CP 2 . 

The hermitian symmetric spaces Gq/Kq are examples of measurable open orbits. 
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In this case, writing the Cartan decomposition as g = t © p, p splits into p + © p_ 
as representations of K. One sees that Gq/Kq is an open orbit in G/KP_. It is 
well known that Gq/Kq is biholomorphic to a bounded domain in some C N . Thus 
the Lie algebra of global holomorphic vector fields is infinite dimensional, so our 
arguments in this case are a little different from the general case. We will treat this 
case first. 

Proposition 2.2. If B = Gq/Kq is a hermitian symmetric space, then Hol(B) is 
finite-dimensional. 

Proof. This is standard. It is contained in Chapter VIII of [7] . 

We now give a condition for Hol(D) to be infinite dimensional. 

Proposition 2.3. If B = Gq/Kq is of hermitian symmetric type and D is an 
open orbit with p_ ^ u then the natural Sbration n : D — > B is holomorphically 
trivial, in particular D = B x K ■ zq. Furthermore, the group of holomorphic 
diffeomorphisms of D is not a finite-dimensional Lie group. 

Proof. Recall the following decomposition of Gq. In the complex group G, P+KP- 
is a dense open set and Go C P+KP-. Furthermore, the decomposition of g G Go 
as g = p + kp- is unique. Then the Harish-Chandra embedding maps gKo e B to 
( G Pi, where p + = exp(£). Denote the image of the Harish-Chandra embedding 
by B. Then B is a bounded domain biholomorphic to B = Gq/Kq. 

Now suppose that D has p_ C u. Then L C Kq and LU C KP-. It is clear 
that D = Gq ■ z C P+K ■ z . The latter is biholomorphic to p + x K /(K D L ). 
To see this, note that the map $ : p + x K /(K Q n L ) — > P+K ■ zq given by 
= exp(£_|_)/c • zq is complex analytic. It is clearly onto. It is one-to- 
one: if exp(£)/c • zq = exp(^')/c / • zq, then by the uniqueness of the Harish-Chandra 
decomposition, exp(£) = exp(£'); so £ = Then k ■ zq = k' ■ zq. We must 
determine the inverse image of D under <fr. Note that elements of Go can be written 
as g = ex.p(£)kp- with £ <E B. Thus, g ■ zq = exp(£)/c • zq. As $ is bijective, 
= i3 x K ■ zq. But K • zq = K ■ z Q ^ Kq/(Kq D Lq). This proves the first 
part of the proposition. 

As B is a bounded domain in some C^, the space of holomorphic functions 
/ : B — > t is infinite dimensional. Now, K • zo = Kq • zo, so if is contained in the 
automorphism group of Kq ■ zq. Thus, for each holomorphic / : B — > t, there is an 
automorphism of B x Kq ■ zq defined by z) = (£, (exp(/(£)))z). Note that 1^0 
since p_ 7^ u. This provides an infinite-dimensional family in Hol(D). □ 

Proposition 2.3 is (1) of the main theorem, since there exists a holomorphic 
fibration D — > £? as in (1) if and only if u properly contains p+ or p_. Also note 
that B is simply connected, whence D — > S cannot have discrete fibers. 
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Assume from now on that u does not contain p_ or p + . In these cases, as we 
will see, Hol(D) is a (finite-dimensional) Lie group. Now consider the Lie algebra 
of holomorphic vector fields on D, that is, the global sections H (D, T) of the vec- 
tor bundle T of holomorphic vector fields on D. Note that T is the holomorphic 
homogeneous vector bundle for the representation g/q of Q. Since Go acts on D 
and on T, it is clear that H°(D, T) is a representation of Gq. There is considerable 
machinery available to study representations of this type. See, for example, [17] 
and [25]. We will very briefly describe the ingredients that we will need. For any 
finite-dimensional representation F of Q, let JF — > Z be the corresponding holomor- 
phic, homogeneous vector bundle on Z. By restriction, we obtain a homogeneous 
vector bundle on D denoted by T — > D. By [25], the Dolbeault co ho mo logy spaces 
H P (D,J-') are continuous, admissible representations of Go- The Harish-Chandra 
module (i.e., the subspace of .KTo-fmite vectors in H P (D,J-') ) is a cohomologically 
parabolically induced (g, _KTo)- m odule TZ^(F). The definition of TZq(F) is given in 
Definition 1, page 432 of [25] (and in §6.3 of [16], with slightly different conven- 
tions). It follows immediately that TZ^(F) C Hom q (W(g), F) Ko _^ nite . Here, U(g) 
is the universal enveloping algebra of g and the subscript 'Ko-fmite' indicates the 
subspace of K -Rnite vectors. 

Proposition 2.4. If E is a finite-dimensional representation of G , then 
Hom Go (E,H°(D,T))^0 Hom q (E, g/q) ^ 0. 

Proof. If : E — > H°(D,T) is a nonzero Go-homomorphism, then the image of 
(p lies in H°(D,T). But H°(D 7 T) C Hom q (W(g), g/q). So defines a nonzero 
element of Hom fl (£, Hom q (W(g), g/q)) = Hom^E, g/q). 

Proposition 2.5. H°(D, T) is finite dimensional and Hol(D) is a Lie group unless 
Gq/Kq is hermitian symmetric and p + C u or p_ Cu. 

Proof. Fix a positive root system A + C A so that A(u) C A + . (We may choose 
A + = A+lU A(u) with A + ([) an arbitrary positive system for A(Q.) Suppose 
F is a finite dimensional Q = LU representation with trivial U action. Then, 
viewing F as a Q = LU representation with trivial U action, by Lemma 5.15 of 
[17], Hom q (W(g),F) Lon ^ . finite ^ W(g)® q F. Furthermore, (W(g) ^F)^.^ is a 
highest- weight Harish-Chandra module (with respect to A + ). By a result of Harish- 
Chandra (cf. [5], Theorem 1 and Corollary 2, page 761), the only case for which 
there exist infinite-dimensional highest-weight (g, K)-modules is when A + contains 
A(p+) or A(p_). 

Now consider the holomorphic tangent bundle T. The action of U on g/q is 
not in general trivial, so the above does not apply directly. Instead, we form a 
filtration g/q = F\ D Fi D ••• D Fn D so that Fi/Fi+i does have a triv- 
ial u action. (For instance, we could take a composition series for g/q, so that 
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each Fi/F i+ i is irreducible as q-module, in which case the action of u is nec- 
essarily trivial.) As rlom q (U(g), • )(L nK )-finite is exact, there is a filtration of 

LqCiKq)- finite with quotients Hom q (W(0), (-^/-^+l))(LonK )-finite — 

(U(g) ®q- (Fi/Fi+i)). Now we may conclude that, unless A + contains one of A(p±), 
Hom q (ZY(g), 0/q)K o -finite is finite dimensional. 

Now note that as we were free to choose A+([) C A, it follows that A(p±)nA([) = 
0. Thus, A + contains one of A(p±) if and only if A(u) contains one of A(p±), that 
is, if and only if p+ C u or p_ C u. 

That Hol(D) is a Lie group now follows from Theorem 3.1 of [9]. □ 

By Proposition 2.5 we may use Proposition 2.4 to restrict the possibilities for 
Hol(D). Note, it is clear that 

(2.6) Hom q (£, /q) 

is non-zero for E = g and, of course, that G C Hol(D). 

Corollary 2.7. If (2.6) is zero for all irreducible representations E j£ g, then 
Hol(D) = G or G. 

Proof. The Lie algebra of Hol(D) lies between g and g. As q is simple the only 
possibilities for Hol(D) are Go and G. □ 

We look for finite-dimensional representations E other than E = g such that (2.6) 
is non-zero. We start off with a fact about the structure of simple Lie algebras. 

Lemma 2.8. If g is a simple Lie algebra, then the Weyl group is transitive on roots 
of a given length. Therefore, there are two possibilities: 

(1) There is just one root length and the highest root, which we denote by 7^, 
is the unique A + -dominant root. 

(2) There are two root lengths and there is a unique dominant long root 7^ and 
a unique dominant short root 7 S . 

Proof. This is standard. See, for example, [8], page 53. 

Theorem 2.9. If g is simple and has just one root length and u does not contain 
p + or p_, then 

Go, if Go is non-compact 
G, if Go is compact. 



Hol{D) = 



Proof. Suppose E is an irreducible finite-dimensional representation of Go- Then, 
by Corollary 2.7, it is enough to check that (2.6) is zero unless E = g. Since A + 
contains A(u), the highest- weight space in E is cyclic for q = [ + u. So, a nonzero q 
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homomorphism : E — > g/q maps the highest- weight vector to a (nonzero) weight 
vector in g/q (of the same weight). This weight must be dominant. As there is just 
one root length, Lemma 2.8 says that this highest weight of E must be 7^. Now 
Corollary 2.7 applies. 

In case Go is non-compact, D 7^ Z and G does not act on D. Thus, Hol(D) = Go- 
When Go is compact, D = Z and G always acts on D. □ 

When there are two root lengths in g, then by Lemma 2.8 we have the possibility 
that the Lie algebra of holomorphic vector fields is or g © E la , where E Ja is 
the irreducible finite-dimensional representation of g with highest weight 7 S . In 
connection with this case we have the following lemma. 

Lemma 2.10. Let g be a simple Lie algebra with two root lengths. Then there 
is a simple Lie algebra g 1 containing g as a subalgebra so that, as g-modules, 
q 1 = q © E ls . The following table lists the possibilities: 






1 


B n 




C n 


A-2n-l 


F 4 


E 6 


G 2 


B 3 



Table 2.11 



Proof. This is easily checked by direct calculation. 

§3. We now consider the simple Lie algebras g having two different root lengths. We 
use Lemma 3.1 below to restrict the possibilities for Z = G/Q for which Hol(D) 7^ 
Go or G. We then treat each of the possible flag manifolds Z separately. 

The following lemma will help us determine when (2.6) is zero for E = E ls . 

Lemma 3.1. Suppose there are roots /3i,...,/3 m G A + , not necessarily distinct, 

so that 7 S — Y^!j=i Pi JS a root f° r eacn ^ = 1> ' ' ' > 171 an d Is — Sj=i Pj JS a ^ on § 
root in u. Then Hom q ( E Js , g/q) = 0. 

Proof. Suppose cj) G Hom q (i? 7s , g/q) is nonzero. If v+ is a highest weight vector in 
E ls then <j>(v+) is a root vector X la of weight 7 S . Since weights of E la all have 
norm less than or equal to || 7 S ||, the long root 7 S — YlT=i Pi * s no ^ a we ight- Thus 
X_p 1 ■ ■ • X_p m ■ v + = 0. On the other hand, ad(X_ / g 1 ) • • • ad(X_p m ) • X ls 7^ in 

g since 7 S — X)j=i Pi ^ s a ro °t f° r eacn k = 1, • • • , m. Since the root 7 S — YlT=i Pi 
is in u, ad(X_ / g 1 ) • • • ad(X-p m ) • X la 7^ in g/q. But this is a contradiction; = 
cf>(X_ Pl ■ ..X_ Pm ■ v+) = ad(X_ /3l ) • • •ad(X_ A J • X ls ^ 0. □ 



8 EDWARD G. DUNNE AND ROGER ZIERAU 

For the simple Lie algebras with two root lengths, we number the simple roots 
as follows: 



The fundamental weight corresponding to otj is denoted by A j . Recall that a weight 
A determines a parabolic subgroup Q with Lie algebra q = q(A) by A(f), q) = {a G 
A | (A, a) < 0}. Each parabolic subalgebra is conjugate to some q(A), in fact, is 

conjugate to one with A = Yl A? f° r some $ C {1, 2, . . . , rank g}. 

je<s> 

Theorem 3.2. Hol(D) is G if Go is compact and is Go if Go is non-compact, 
except possibly in the following cases: 

Type B n : Q is determined by X n , 
Type C n : Q is determined by Ai 
Type G2: Q is determined by \\. 

Proof. To apply Lemma 3.1 we find, for each simple Lie algebra having more than 
one root length, roots j3j as in the lemma so that (Aj, 7 S — Y^?=i Pj) > f° r an -\j 
except those listed in the theorem. 

For type B n , 7 S = Y^=i a j an d 7s — a n is a long root. (So m = 1, j3\ = a n in 
Lemma 3.1.) Clearly (Aj, 7 S — a n ) > except for j = n. 

For type C n , 7 S = ai + 2a 2 + 2«3 + • — h 2a n _i + a n and 7 S — ai is a long root. 
(So m = 1, Pi = ol\ in Lemma 3.1.) Clearly (Aj, 7 S — cti) > except for j = 1. 

For type G2, 7 S = 2cti + «2 and 7 S — 2a±(= 0^2) is a long root. (So m = 2, j3\ = 
02 = ct2 in Lemma 3.1.) Clearly (A2, 7 S — 2«i) > 0. 

For type F 4 , 7 S = 2a\ + 3ct2 + 2a 3 + ct 4 and 7 S — «2 is a long root. (So m = 
1, (3i = a.2 in Lemma 3.1.) Clearly (Aj, 7 S — 0^2) > for all j. Thus, every parabolic 
subalgebra of F 4 satisfies the condition of Lemma 3.1. □ 



We now prove our main theorem by looking at each flag manifold listed in The- 
orem 3.2. 
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Type B n . Let p, q be positive integers such that p + q = n. The complex group 
G = SO(2n + 1, C) is defined by the symmetric form on C 2n+1 : 



The relevant real form Go of B n is the connected component of the isometry group of 
( , ) restricted to R 2n+1 . That is, G = SO e (2p, 2q + 1). The complex flag manifold 
Z is the space of all maximal isotropic subspaces of C 2n+1 , also known as the space 
of pure spinors. The set of (p + (/)-planes £ G Z such that the hermitian form 
(w, z) := (w, z) restricted to ( has signature (p, q) contains two open Go-orbits. 
They are D± = Gq-Zq, where z^ = span c {ei ± ie 2 ,e 3 + ie 4 , . . . ,e 2n -i + ie 2n }- 
These are the only open orbits of Go in Z. 



Let G 1 be the isometry group of (w, z) 1 = J2j=i w j z j~Yl,j=2p q +i w j z j on C 2n+2 . 



Let 67q be the connected component of the isometry group of ( , ) 1 restricted to 
R 2n+2 . Then, Gq = SO e (2p, 2q + 2). The space of maximal isotropic subspaces in 
C + has two connected components (since the dimension is even). Take Z 1 to be 
the component containing Zq = spanc{ei + ie 2 , • • • , e 2n _|_i + ie 2n + 2 }. (The other 
component contains spanc{ei +ie 2 , .. . , e 2n _i +ie 2n , e 2n _|_i — ie 2n+2 }.) Then Z 1 is 
a flag manifold for G l . Again, there are two open orbits D± 1 consisting of planes 
of signature (p,q + 1) with respect to the hermitian form (w, z) 1 := (w,^) 1 . 

Define a map n : Z 1 -> Z by n(() =(n (C 2n+1 x {0}). Note that tt(C) G Z as 
it is clearly isotropic and dim(7r(C)) = n. It is also clear that tt is Go-equivariant. 
By equivariance, n is onto. To see that n is one-to-one it is enough to check that 
Tr- 1 ^) = {zq}. Suppose tt(C) = C n (C 2n+1 x {0}) = z . Then 2 C C and 
there is v G ( so that v G ^o ±(,) H ^o" 1 * = (zo +^o)" L and C = z o © C • v. But 
(zq + Z0) 1 - = span c {e 2n+ i, e 2n+2 }. As v is isotropic it is either e 2n +i + «e 2n+2 or 
e 2 n+i - ie 2n +2- Only v = e 2n +i + ie 2n +2 gives ( G Z 1 , so C = Zq. 

We may conclude that G acts transitively on Z 1 = Z and Gq acts transitively on 
D± = .D±. Equivalently, G 1 and 67q act on Z and Z)±, respectively. It also follows 
that the compact real form acts on the Go-orbit Z. 

Type C n . Let u(w,z) = YLl=i( w j z n+j ~ w n+jZj) be the standard symplectic 
form on C 2n . Then G = Sp(n, C) is the complex group preserving uj. The complex 
flag manifold under consideration is Z = {cu-isotropic lines in C 2n }, which is just 
CP 2n_1 , since any line is automatically isotropic. There are two families of real 
forms: Sp(n, R) and Sp(p,q). 

Define Go = U(n, n) H G, where U(n, n) is the isometry group of (w, z) = 
Y^j=i( w jZj ~ Wn+jZn+j). Then Go — Sp(n, R). There are two open Go-orbits, 
D±: the positive lines and the negative lines. But, it is clear from Witt's theorem 
that Gq = SU(n,n) acts transitively on D±. 



2p 



2p+2q+l 




j=2p+l 
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For the other real forms, take Go = U(2p, 2q)(~)G, where U(2p, 2q) is the isometry 
group of (w,z) = YHj=i( w jZj + w n+j z^-) - Yl P jtl+i( w jZj + ^n+j^+j), where 
p + q = n. Then Go = Sp(p, q), (cf. [7], p. 445). For pq ^ 0, Go has two open orbits 
consisting of positive and negative lines. Also, Gq = SU(2p, 2q) acts transitively 
on each of these orbits. If pq = 0, then Go is compact and acts transitively on 
Z = CP 2n_1 . The unitary group SU(2n) also acts transitively on Z. 

Type G 2 . Let G be the complex group of type G 2 and let G 1 be the complex 
group of type B3. Let Go and Gq be the split real forms of G and G 1 , respectively. 
We first recall how Go C Gq and G C G 1 . The split real form Go of type G2 is the 
automorphism group of the split octonions (i.e., the Cayley numbers), O. There is 
a natural inner product of signature (4, 4) defined on O. It is easy to see that any 
automorphism must preserve the inner product. Since an automorphism must fix 
1 G O, it must preserve ImO := (R- l)- 1 , the pure imaginary octonions. Moreover, 
an automorphism is completely determined by its restriction to Im O. We see, then, 
that G C SO e (3,4) ^ G\. Complexifying, we obtain a complex symmetric form 
( , )c on (ImO)c- Similarly, G sits inside the isometry group G 1 of ( , )c- There 
is a corresponding hermitian form defined by (w,z) := (w,z)c- 

Consider the flag manifold for G 1 defined by Z = {isotropic lines in (ImO)c — 
C 7 }. This is a flag manifold for G under the action of G as a subgroup of G 1 . The 
proof of this is similar to the proof of Claim 3.5 below and is sketched at the end of 
this section. We will see that Gq = SO e (3, 4) acts on Z with two open orbits D± 
and that Go acts transitively on both. 

The action of Gj = SO e (3, 4) has two open orbits, _D±, respectively, the positive 
and negative lines (with respect to ( , )) in Z. We verify this as follows. Write 
z = x + iy G (ImO)c = ImO + zlmO with x,y G ImO. Then z is positive and 
isotropic if and only if: 

(3.4) (x, x) = (y, y) > and (x, y) = . 

Suppose z' = x' + iy' is another positive isotropic vector. By scaling z, we may 
assume (x,x) = (x',x') and (y,y) = (y',y')- Now, Witt's theorem says that there 
is an isometry of ImO = R 3 ' 4 taking x to x' and y to y', i.e., taking z to z' . One 
can check that this isometry can be chosen to lie in SO e (3,4). 

Claim 3.5. G acts transitively on D±. 

We use the following lemma, which follows easily from the development in Chapter 6 
of [6]. 

Lemma 3.6. Suppose A, A' C O are normed subalgebras both isomorphic to 
either the quaternions H or the 2x2 reai matrices M2(R), then any isomorphism 
A — > A' extends to an automorphism of O. 
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Proof. Choose e E A 1 - and e' E (A') 1 - with (e, e) = -1 = (e',e'). Then, by 
Lemma 6.15 of [6], A 1 - = At and O = A © Ae. Moreover, multiplication in O 
is given by 

(3.7) (a + be) (c + de) = (ac + db) + (da + bc)e, 

where the bar denotes conjugation in A: the usual conjugation for H and 

fa P\ = ( & ~P\ 
\7 8 J \~1 ol ) 

for M 2 (R). Similarly for O = A'®A'e'. Now suppose / : A — > A' is an isomorphism. 
Then, as one can readily check, F(a + be) := f(a) + f(b)e' is an automorphism of 

6. □ 

Remark 3.8.. In case A = A', F e >(a + be) := f(a) + f(b)e' extends / to an auto- 
morphism of O for any e' E A 1 - with (e',e') = —1. In fact, any automorphism of 
O that preserves A must send e to such an e'. Therefore, all extensions of / are of 
this form. 

Proof of Claim 3.5. If z and z' are two elements in D + with z = x + iy and 
z' = x' + iy', then both decompositions must satisfy (3.4). By Proposition 6.40 of 
[6], the subalgebras A and A' of O generated by {x,y} and {x',y'} are isomorphic 
to H, say, : A -> H and 0' : A' -> H. Then, / = (J)' 1 o 0' : A' -> A is an 
isomorphism from A' to A. 

Now, both x, j/ and f(x'), f(y') satisfy (3.4). By rescaling z', if necessary, we 
may assume (x, x) = (f(x'), f(x')). Also, Aut(H) consists solely of the conjugations 
by unit quaternions, which is the map SU(2) — > SO(3), acting on Im(H) = R 3 . By 
Witt's theorem, there is an orthogonal map h (i.e. an automorphism of H) sending 
f(x') to x and f(y') to y. Then, h o f extends to an automorphism g of O, by 
Lemma 3.6. Thus, we have an element g E Go with g(z') = z. 

For z, z' E D_, the argument is similar. Now, however, the subalgebras A and 
A' generated by {x, y} and {x',y'} are isomorphic to M 2 (R). Again, there exists 
an isomorphism / : A' — > A. The group Aut(M 2 (R)) consists solely of conjugations 
by elements of SL(2, R), which is the action of SO e (l, 2) on R 1 ' 2 ^ M 2 (R) n ImO. 
Again, by Witt's theorem, there is an isometry h of R 1 ' 2 such that h(f(x')) = x and 
h(f(y')) = y. One can check that h may be taken to lie in SO e (l,2) = SL(2, R). 
Now, by Lemma 3.6, hof extends to g E Aut(O). Thus, we have an element g E Gq 
with g ■ z' = z. This completes the proof of Claim 3.5. □ 

Remark 3.9. As homogeneous spaces, the two open orbits are D+ = 67o/U(2) and 
D_ = 67 /(SL(2, R) x U(l)). This is a consequence of Lemma 3.6 and Remark 3.8, 
as we now demonstrate. 
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For D + , write A = H = ® J=1 Re.,- with e\ = identity and e2,e3,e4 units (i.e. 
(ej,ej) = 1) with e-i es = e^. Suppose g E Go fixes e2 + ies G D + . Then g fixes 
both e2 and es- As g is an automorphism, it must also fix e^, hence all of A. 
By Remark 3.8, the automorphisms fixing A are of the form F ae , where a is an 
element of A of length one. But, since A = H, the length-one elements form a 
group isomorphic to SU(2). The stabilizer of the line C • [ei +ies) also includes the 
scalars. Therefore, Stabc (e2 + ies) — U(2). 

For the orbit _D_, take A = M 2 (R). We pick our base point to be the isotropic 



then g fixes A. By Remark 3.8, the automorphisms of O fixing A are all of the form 
Fp e where (3 G A and (/?, /?) = 1, i.e., det(/3) = 1. Thus, the group of automorphisms 
fixing the vector x + iy is SL(2, R). As scalars also fix the line C • (x + iy) , we have 
Stab Go (x + iy) = SL(2, R) x U(l). 

We conclude the type G2 case by sketching a proof that Z is a flag manifold for 
G. For this it is enough to show that a compact real form of G2 acts transitively 
on Z. To do this we use a slightly different realization for G. A compact real form 
of G2 is the automorphism group of O, where O is the nonsplit octonians. There 
is a positive definite symmetric form on O preserved by the automorphism group, 
Aut (O). As above Aut (O) preserves ImO = (R-l) ± , thus Aut (O) C SO(7). Now 
complexify the form to obtain a symmetric form on (ImO)c- The isometry group 
is conjugate to G 1 (as the form is equivalent to the complex form arising from the 
split octonions). The corresponding flag manifold is the space of isotopic lines in 



Now let z = x + iy and z' = x' + iy' be isotropic vectors in (Im 0)c- Then (3.4) 
holds for both z and z' . The subalgebras A, A' generated by {x,y} and {x', y'} 
are both isomorphic to H. Now argue as in the case D + using the fact, analogous 
to Lemma 3.6, that an isomorphism A — > A' extends to an automorphism of O. 

Part (3) of the main theorem now follows immediately since the groups G or G\ 
preserve the metric and the corresponding complex groups do not. This completes 
the proof of the main theorem. 

Now suppose that G is semisimple. Write q = gi © ■ • • © Qn with 0i, . . . ,0at 
simple and let Gi be the subgroups of G corresponding to the gi. A flag manifold 
for G is the product of flag manifolds Z t = Gi/Qi for the Gi. A measurable open 
Go-orbit D is the product of measurable open G^o-orbits Di in the Z^. Recall that 
by Proposition 2.3, writing = U + Ui, if 



then Hol(Di) is not a (finite-dimensional) Lie group. Thus, if at least one of the 
factors Di satisfies (3.10) then Hol(D) is not a Lie group. 




ImO. 



(3.10) 



Pi 



C Ui or p i}+ C Ui 
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Proposition 3.11. If no factor Di satisfies (3.10) or is a hermitian symmetric 
space then Hol(D) ^ Hol{D x ) x • • • x Hol{D N ). 

Proof. By the same argument as in 2.5, Hol(D) is finite dimensional. We look 
for representations E such that Hom q (E, jj/q) ^ 0. The only such irreducible 
representations E are E = Qi and E = g], the q} occuring when Zi appears in 
Table 1.1. It follows that Hol(D) = Hol(Di) x • • • x Hol{D n ). 

§4. The main theorem has several consequences in representation theory. As men- 
tioned earlier, interesting representations are those occurring as Dolbeault cohomol- 
ogy of a line bundle over an open orbit D. These are the representations associated 
to elliptic co-adjoint orbits by the orbit method. See for example [14], [15], [16] 
and [18] for some of the important properties of these representations. Suppose we 
have a holomorphic homogeneous line bundle C x — > Go/L corresponding to the 
character of L whose differential is x G Uq. If 

(4.1) ( x + p,p)<0, for all (S e A(u) 

(where p is half the sum of the positive roots) then the Dolbeault cohomology space 
H p (Go/ 'Lq, C x ) = for p 7^ s = dimc(i^o • zq) and H s {Gq/Lq, C x ) is an irreducible 
admissible representation. When Gq is non-compact the main theorem gives the 
cases where Hol(D) = Gq~ ~jb Gq. From the case-by-case descriptions given in Sec- 
tion 3, it is clear that D = G /L ^ G^/Lj is an open orbit in a flag manifold for 
G 1 . The character defining C x extends uniquely to a character of Lq whose differ- 
ential we will call x 1 . Thus the bundles C x — > Gq/Lq and C x i — > G\/L\ are holo- 
morphically and Go-equivariantly equivalent, implying the Dolbeault cohomology 
spaces are Go-isomorphic. Thus, in the range where (4.1) holds, the representations 
H s (Gq~/Lq, C x ) are irreducible representations of G\ which remain irreducible when 
restricted to Gq. Put differently, the irreducible representations H s (Go/L , C x ) of 
G extend to representations of Gq. This is a somewhat rare phenomenon. Wolf 
[20] obtained some general results on the irreducibility of a representation when 
restricted to a subgroup, which have some overlap with this application of our main 
theorem. Also, see [10] for results on restricting representations of this type. More 
recently, A. Dvorsky has a number of new results along these lines. 

A well-known and important example of the phenomenon discussed above occurs 
in quantum electrodynamics. A family of "massless" representations of the de Sitter 
group (SO(2, 3)) extend to representations of the conformal group (SO(2, 4)), cf. [1]. 
As these representations can be realized naturally in Dolbeault cohomology (for the 
open orbits in the first entry of Table 1.1 for p = 1 and q = 1), the geometric 
explanation for this extension is an instance of our main theorem. 

When Go is compact, for arbitrary x, H p (Gq/L , C x ) = H p (Gq/Lq, £ x i) as 
G representations for all p. By the Bott-Borel-Weil theorem we see that certain 
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representations of Gq remain irreducible when restricted to Go- The following table 
gives the highest weights x an d X 1 of these representations. 



1 


x 1 


Q 


X 


Dn+1 


a\ n+ i 


B n 


aX n 


A2n-1 


aXi 


C n 


aXi 


B 3 


aXi 


G 2 


aXi 



a G Z + 
Table 4.2 



Our results also have implications for the study of deformations of maximal 
compact subvarieties in D, as in [4], [19], [24]. In particular, let Vo be the compact 
subvariety Kq ■ zq in D = Gq/Lq, where Go is non-compact and D is, as usual, 
an open orbit in a generalized flag manifold for G. The cycle space studied 
in [19] is the space of translations of Vo by elements of G which remain in D: 
M D = {g ■ V C D\g e G}. The space M D plays a key role in constructing a 
transform, often called a Penrose transform, for the representations H S (D, C x ). In 
most cases, Mo has the dimension that one computes using Kodaira-Spencer theory 
(cf. [11]). However, in some cases, the Kodaira-Spencer theory predicts a larger 
space of deformations. For some of these cases (but not all), our results explain 
the discrepancy by demonstrating that there is a larger group of automorphisms G\ 
acting on D. Compare with [13]. 

For example, when q is of type C n and Go = Sp(n, R) we have seen that the 
space of positive lines in CP 2n_1 is an open orbit. Then Vo = CP n_ . Kodaira- 
Spencer theory predicts that the (infinitesimal) deformations of Vo in D come from 
H°(Vo,Af), where M is the holomorphic conormal bundle of Vo in D. In this exam- 
ple, H°(Vo,Af) = { symmetric nxn matrices} © { skew-symmetric nxn matrices} 
as representations of Kq. Let's denote the deformation space of [19] for the groups 
Go = Sp(n, R) and Gq = SU(n, n) acting on D by Md and M D , respectively. 
One can check that in this case M D = G /K and Mp = G\/Kl (cf. [24]). As 
dimc(Go/i^o) = dim(Go/ifo) + n ^ n ~ 1 ^ , the deformation space of Kodaira-Spencer 
is strictly larger than Mo in dimension. The extra n ^ n 2 ~ 1 - > dimensions come from 
the skew matrices B by the action of 

6XP (^ ^) G u ( n > n ) x Sp(n,R) 

on Vb. Note that the infinitesimal deformations predicted by Kodaira-Spencer the- 
ory all come from M^. 

As a final remark, note that the list in [2] of nilpotent co-adjoint orbits 'sharing' 
an orbit with a bigger group has a lot in common with our list. The open orbits we 
are considering here are Go-equivariantly biholomorphic to the elliptic co-adjoint 
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orbits. It would be interesting to understand the connection between our list and 
that of [2] in terms of nilpotent orbits as limits of elliptic orbits. 
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